More than a Number: Unexpected Benefits of Return on Investment Analysis 

Abstract 

In 2010-2011, University of Colorado (CU) librarians implemented a multi-campus pilot study to measure 
the institutional value of library resources used by faculty in their research. The study incorporated 
quantitative methods including return on investment (ROI), cost benefit analysis (CBA), and citation 
analysis of journal articles published by faculty; and qualitative methodologies such as in-person 
interviews with faculty. The study resulted in a CU ROI model that can be used to measure faculty 
perceptions of value and the economic benefits of electronic journal collections for faculty research in 
terms of ROI. The CU ROI methodology provides outcomes beyond a single ROI number and led to 
unexpected benefits for informing collection development decisions and strategies. 

Introduction 

On Monday, September 29, 2008, the US stock market crashed -- losing 1.2 trillion US dollars in a single 
day (Bajaj & Grynbaum, 2008). No country or industry was immune to such a dramatic collapse. The 
Chronicle of Higher Education stated, "higher education responded in an uncharacteristic way: It began 
to buckle." With no crystal ball, "The crisis will force [campus officials] to make tough choices: Should 
they reduce payouts to protect their endowments or increase them at a time when students are sure to 
need more help? And if they have less to invest in people, programs, and buildings, which should come 
first?" (Wilson, 2008, Squeezing the Middle Class para. 11). Universities have endured economic 
downturns before. However, this financial crisis began to question old assumptions in new ways. In the 
article entitled "Seeking the New Normal: Budget Strains Force Radical Change," Henderson and Bosch 
(2010) assert "[ejven when the economy improves, increased funds for libraries are not likely to be at 
the top of the list for new spending priorities." In this context, the stage has been set for new 
interpretations and definitions of library values and contributions to university missions and outcomes. 

For collection development and acquisitions librarians at the University of Colorado, the economic 
downturn along with increased scrutiny of library materials budgets from deans, directors and 
administrators created an opportunity to investigate ways to demonstrate library value. The University 
of Colorado System (comprised of three institutions, four campuses, and five libraries) has a tradition of 
purchasing electronic materials collaboratively. This institutional library consortium focuses on 
leveraging library budgets to jointly license resources and streamline procurement and legal review. In 
the fiscal year 2010 (July 2009 to June 2010), the collective buying power was over seven million dollars 
and shared e-resources ranged from a single journal title to large publisher packages. To determine the 
return on investment of CU System joint purchases for university administrators, CU librarians 
conducted a cost benefit analysis of combined budgets and also gathered personal accounts and 
anecdotes that described the economic benefit that students and faculty received from consortia 
purchases. The results of this initial study, previously published, "provided compelling evidence for the 
value of collaborative purchasing to library administrators and university officials. In the end, they were 
able to take their consortia purchases 'off the table' for cuts and continue to provide greater access to e- 
resources for at least another year" (Pan & Fong, 2010, p. 190). 

Although CU librarians successfully conveyed positive cost benefit data to management, the ROI 
analysis of consortia spending did not provide direct evidence of how library collections support faculty 
research. The need to answer the question "How is the relationship between library resources and 
scholarly outcomes measured?" became evident in the fall of 2010 when the University of Colorado 
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Denver's Auraria Library was preparing for their academic department program review, as required by 
the CU Board of Regents and the Colorado Department of Higher Education. The self-study portion of 
the review asked the Auraria Library to describe how the relationship between library resources and 
instructional and scholarly outcomes is measured, and how these results can be used to revise and 
strengthen programs. Given time limitations, traditional and readily available library data was used for 
the program review, but these experiences prompted the authors to develop the CU ROI pilot study 
(Pan, Williams, Wiersma, & Fong, 2012). 

The CU ROI pilot study attempts to measure the impact of library collections on faculty research 
outcomes by determining if the library was able to provide access to the electronic journals faculty 
needed for their publications. The investigators adopted a shared research protocol that incorporated 
both quantitative and qualitative methods, including citation analysis of published journal articles and 
content analysis of faculty interview questionnaires. In order to measure a return on investment, the 
authors assigned a monetary value to journal articles used in research, identified library subscription 
costs, and developed a formula for determining faculty benefits from library collection expenses. In the 
context of experiences at CU Boulder, Denver, and Anschutz Medical campuses, the authors will provide 
a literature review of library value studies; describe their procedures for articulating values, developing 
metrics, and calculating results of the CU ROI model; and highlight examples of maximizing indirect 
benefits from the ROI process. 



LIBRARY VALUE STUDIES 

Since the early 1990s, libraries have used ROI/CBA to measure the value of their collections and services. 
Economic pressures have motivated public libraries to conduct valuation studies to demonstrate how 
taxpayer funding benefits citizens and communities. Imholz and Arns (2007) provide a summary and 
assessment of 17 public library valuation studies completed from 1998 to 2007. They argue that 
"cost/benefit analysis appears to be the most effective language for conveying the efficiency of 
investments of tax dollars in public libraries to public constituents" (Ibid., 2007, p. 45). Covering a similar 
time period, 1995-2008, Aab0 (2009) offers an international meta-analysis or quantitative evaluation of 
previous empirical studies. Findings show that 84.2% (32 out of 38) studies focus on public libraries 
world-wide. With this sample size, Aab0 (2009) states, "...a tentative conclusion can be drawn. The 
results shown in these studies indicate that for each dollar of taxpayers' money invested in public 
libraries, the libraries - on average - return a value to the citizens of four to five times more" (p. 322). 

In addition to justifying budgets, public library impact studies have applied ROI/CBA analysis to other 
circumstances. Matthews (2011) identifies three additional uses of ROI/CBA analysis: to prioritize 
alternative and scarce resources; to evaluate an existing project or service; and to support a decision 
quantitatively. He cautions, however, "[rjelying on a single measure, such as Return on Investment, is 
not likely to produce a positive reaction among the library's funding decision makers year after year" 
(Ibid., 2011, p. 11). Simply stated, there is "no silver bullet." Rather, libraries may be more continuously 
successful by establishing an "overall framework for displaying and reporting several key performance 
measures to be effective among decision makers and stakeholders. The library's ROI could be one of the 
key performance measures" (Ibid.). Both Matthews (2011) and Holt (2007) describe the indirect benefits 
or values public libraries provide to their patrons. For example, based on thousands of interviews 
conducted during a CBA study, Holt (2007) identified eight services valued by public library users: time 
savings; financial savings; convenient location reducing traveling time; access to materials; assistance by 
library staff; services for children; access to technology; and provide good value for taxes paid. 
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Responding to similar economic pressures as public libraries, medical libraries have been conducting 
reports and developing tools to measure impact with ROI and CBA. For example, a work group of 
Department of Veterans Affairs (VA) hospital libraries recognized the need to develop a standardized 
method of measuring return on investments. The VA Library Network Librarians' approach includes 
three components: Return on Investment Analysis tool; a Library Scorecard; and Management Support 
Report (Jemison et al., 2009). Furthermore, in 2008 and 2011 the Health Sciences and Human Services 
Library (HS/HSL) of the University of Maryland used an automated tool developed by the National 
Network of Libraries of Medicine (NN/LM), Mid-Continental Region, to justify their annual book and 
journal expenditures (Bodycomb & Del Baglivo, 2012). A majority of these public and medical library 
analyses focus on quantifying the benefits of library services for patrons. 

More recently, academic libraries are responding to the growing need to connect library services and 
collections with institutional outcomes. Mezick (2007) developed a study to demonstrate how 
expenditures on library resources and staff services contribute to student persistence and retention. Her 
research was based on data provided by the Association of Research Libraries (ARL), Association of 
College and Research Libraries (ACRL), and Integrated Postsecondary Education Data System (IPEDS) of 
the National Center for Education Statistics (NCES). Mezick (2007) asserts that "... while statistically 
significant relationships exist between each category of expenditure and student retention within every 
Carnegie Classification, the strongest relationships exist between total library expenditures, total library 
materials expenditures, and serial expenditures at baccalaureate colleges" (p. 564). Creaser and Spezi 
(2012) conducted eight case studies using interviews and surveys in Scandinavia, USA, and the United 
Kingdom to study the value of academic libraries for teaching and research staff. Their research revealed 
"more evidence of activity rather than evidence of value and impact, especially value to and impact on 
teaching and research staff" (p.l). 

While public and special libraries were early adopters of ROI, academic libraries have also used ROI to 
demonstrate value. Tenopir and King (2007) interviewed faculty to determine the quality and value of 
journal article readings. The study found that having access to electronic journals saved faculty time and 
increased the number of articles they read over time. As a result, they calculated "ROI in the journal 
collection made by the university" by estimating how much time and money was saved by having access 
to the library's e-journal collection (p. 204). 

In 2008, the University of Illinois at Urbana-Champaign (UIUC) produced a major study using ROI/CBA to 
measure the value of library collections, focusing on income generation through grants as opposed to 
cost savings. According to Luther (2008), the case study was framed with an economic perspective and 
"using the ROI model with UIUC data produced a return of $4.38 in grant income for every dollar 
invested in the library in 2006" (p. 4). Over the next couple of years, the ROI model created in the UIUC 
case study was replicated in eight institutions and in eight countries (Tenopir, 2010b). Tenopir (2010a) 
found that the mission and subject emphasis of the institution impacted the ROI of e-journal values. She 
explains the variation in terms of "the purpose of the institution, with the high being for a pure research 
institute and lower for teaching/research universities in countries without a high number of competitive 
grant funds... [and] subject discipline, which may account for the differences between institutions 
depending on the degrees they offer or relative size of subject disciplines" (p. 45). Furthermore, she 
warns of relying solely on ROI for measuring the value of library collections and services. Similarly, 
Sidorko (2010) voices his concern with the complexities of individual countries and institutions, data 
gathering process, and using grant funding as the basis of demonstrating ROI. He articulates the need to 
"highlight this growing need and to move some way towards developing a methodology, or more likely, 
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methodologies, that can be adapted contextually to suit institutional individuality and idiosyncrasy" (p. 
652). 

Likewise, publishers have recognized their vested interest in helping libraries identify products and 
services that provide the most value to the university community and best support university mission 
and goals. For example, the research and advisory firm Outsell prepared the report "Establishing Value 
and ROI: Investing in STM E-journals and E-books" at the request of Springer and Loughborough 
University was commissioned by SAGE to write "Working Together: Evolving Value for Academic 
Libraries" (Hydock, McShea, & Ohri, 2009; Creaser & Spezi, 2012). The Outsell report further explains 
that, "[assessing and demonstrating the value of paid information is a top issue for libraries and 
information centers today, with most information managers facing a real struggle to put together all the 
pieces needed to demonstrate the true worth of information and the return on investment (ROI). ..Value 
is inherently subjective, with different assessments from one individual to another, from one user group 
to another, and across an organization" (Hydock, McShea, & Ohri, 2009, p. 5). 

Several additional initiatives have emerged to measure the contribution of academic libraries. Some 
notable examples include: the Lib-Value Project funded by a three-year grant from the Institute of 
Museum and Library Services awarded to the University of Tennessee, Knoxville, in collaboration with 
the University of Illinois at Urbana-Champaign Libraries and the Association of Research Libraries (2010); 
the Library Assessment Conference sponsored by ARL, the University of Virginia Library, and the 
University of Washington Libraries (2012); and The Value of Academic Libraries: A Comprehensive 
Research Review and Report developed for the Association of College & Research Libraries by Oakleaf 
(2010). Strong participation by institutions and associations demonstrates the importance of identifying 
and implementing library value studies. 



Developing The CU ROI Pilot Study 

Each of the three CU campuses is different in size, mission, graduate programs, student population and 
resource allocations. Recognizing these differences the researchers sought to develop a consistent 
methodology for the pilot study that could be applied to each institution regardless of size or mission. 
The CU ROI model was largely inspired by the research at the University of Illinois at Urbana-Champaign 
which used the total dollars spent on library collections and services as the costs and the dollars 
returned in sponsored research awards as the benefits (Luther, 2008, p. 8). The CU ROI model uses the 
library's journal subscription expenses as the costs and the average price that faculty would have to pay 
to purchase an article, if the library did not subscribe, as the benefits (Pan, Williams, Wiersma, & Fong, 
2012). The study also builds upon Tenopir's suggestion that "ROI calculations should be expanded 
beyond grant income to include the value of all key library products and services that support the 
mission of the institution" (Tenopir, 2010b, p. 3). 

The pilot study consists of four overarching phases: 1) defining value, 2) gathering data, 3) calculating 
ROI, and 4) analyzing results. In addition to quantifying a value that can be tied to scholarly outcomes, 
each phase of the process can provide opportunities to discover insights such as improving or 
establishing a relationship with a department and individual faculty members; promoting library 
services; and identifying highly valued journals, patterns of faculty use, and alternative serials 
purchasing options. 
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Phase 1: Defining value 



Many methods exist for defining and measuring library value. According to Oakleaf (ACRL 2010), 
libraries most often define value through "use" or utility, and financial value. "Use" focuses on counting 
inputs and outputs like circulation statistics, full-text downloads, or citations. Counting usage measures 
"implicit value" by equating use with value and concludes that libraries are valued because they are 
used (Tenopir 2007, 2011). While informative, usage statistics "do not show why someone used or 
requested a source or the outcomes on their work from using that source" (Tenopir, 2009, p. 10). 
However, they can be paired with explicit values which seek to capture why the library collections are 
important to users. Explicit values can be provided by observing or interviewing users about their use of 
library collections and identifying what they produced or achieved as a result (ACRL 2010, Tenopir 2011). 
Combined with implicit values, "qualitative methods can add to [the] story of the value and outcomes of 
library collections and services" (Tenopir, 2011, p. 9). Finally, financial value is often based on a formula 
of dividing perceived benefits by perceived costs to determine library value. Costs and benefits may be 
defined as prices, time, effort provided by library or possible exchange value received by patrons from a 
library service or resource. Financial value can be presented as a cost benefit analysis or expressed as 
return on investment. 

The CU ROI methodology combines multiple definitions of value to measure the impact of library 
collections on faculty research. The authors initially selected return on investment and cost benefit 
analysis as financial value metrics that would appeal to academic administrators. However, as Tenopir 
(2011) explains, ROI is a "derived value" that requires "multiple types of data collected on both the 
returns (benefits) and the library and user costs (investment) to explain value in monetary terms"(p.6). 
The CU ROI pilot study measured use of library collections by counting journal citations in faculty 
articles. Unlike generic usage statistics which are disconnected from scholarly outcomes, counting cited 
reference signify use of a source that resulted in a published article. In addition, faculty authors were 
interviewed to determine how they acquired scholarly information for their research. Faculty at each 
campus indicated that the majority of the resources they used in their research were obtained from 
library sources. The authors positioned financial value in terms of the library's costs to subscribe to 
journals cited compared to the pay-per-view costs for the same cited articles. As outlined in the 
following phases, the study led to the development of a ROI model that incorporated faculty 
perceptions of value and the economic benefits of electronic journal subscriptions for supporting faculty 
research. 

Phase 2: Gathering Data 

The next phase of the study required identifying, recruiting, and interviewing faculty. The goal of the 
interviews was to gather curriculum vitae and information on the project participants' use of library 
materials. While the investigators considered using participants from the same discipline on all three 
campuses, time constraints made that approach difficult to achieve. Instead, each campus followed 
similar protocols and identified potential participants as academic faculty with research responsibilities. 
Faculty members with appointments that focused on teaching without a research requirement were 
removed from the participant pool. Researchers from all three campuses consulted with subject liaisons 
to identify academic departments or individual faculty members who would be willing to join the study. 
Both the Denver and Anschutz Medical campuses concentrated on a single academic department, the 
Chemistry and Physical Therapy departments respectively. In contrast, the Boulder study recruited 
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participants from a variety of departments, such as Engineering, Business, History, Philosophy, 
Sociology, Economics and Asian Studies. 

At the Denver campus, Pan conferred with the Science Collection Development Librarian, and also met 
with the Dean of the College of Liberal Arts and Sciences and the Chemistry department chair to explain 
the project in person. Pan mentioned their endorsement and carbon copied the department chair on an 
email invitation to each full-time faculty member in the department which persuaded nearly 70% (9 out 
of 13) to participate (Pan, Wiersma, & Yem Fong, 2011). Similarly, at the Anschutz Medical Campus, 
Williams and an academic liaison obtained support from the Physical Therapy program director, and 
recruited 57% (12 out of 21) of the department (Williams, Pan, Wiersma, & Fong, 2012). At the larger 
research-focused Boulder campus, Fong and Wiersma enlisted the support of the Boulder Faculty 
Library Committee, a faculty committee of representatives from across campus units, and the Dean of 
Libraries, who sent an email to all faculty members inviting their participation. Fong and Wiersma also 
asked subject liaisons to actively recruit participants through their departmental faculty contacts and 
invited liaisons to sit in on the interviews (Wiersma, Pan, Williams, & Fong, 2012). Even though the 
researchers were not able to interview a representative sample of the faculty on each campus, the pilot 
study explored several methods to approach and recruit faculty that could be used to complete a more 
comprehensive study in the future. 

During the interviews, the researchers described the goals of the project; requested a copy of each 
participants's curriculum vita and syllabi for the citation analysis; and asked 4 questions about their use 
of library resources. 

1. In your most recently published article, for every article you cited how many additional articles 
did you read but did not cite ? 

2. Approximately what percent of the articles cited or read were obtained from the following: 

- electronically from the library? 

- in print format from the library? 

- via Interlibrary loan from the library? 

- from non-library sources? 

Faculty were asked to describe their most recently published article and describe how many of the 
resources cited in that publication came from library or other sources. However, these encounters also 
presented an opportunity for the librarians to gain feedback on existing collections and services. Similar 
to Holt's (2007) study, faculty participants at each campus identified resources and services of value to 
their research and teaching. 

For example, the Boulder faculty who participated in the pilot study reported differences in the use of 
electronic journals and differences in collection needs based on discipline. For faculty in the humanities, 
although access to journal literature was important, they were more interested in discussing access to 
primary sources, monographs and reference literature. Social science and science faculty were much 
more concerned about maintaining access to key online journals and databases in their fields. Boulder 
faculty uniformly praised the University Libraries' interlibrary loan services and noted that the transition 
to e-books was useful for teaching. The interviews also generated discussions about interdisciplinary 
needs and the complexities of a recent journal cancellation project. Since several subject specialists sat 
in on the interviews, these conversations created greater relationship building opportunities with their 
faculty. In addition to answering questions about how to access library collections, the librarians were 
able to promote a new document delivery service and offer instruction sessions for students. 
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While Boulder researchers were collection development librarians who typically have on-going 
interactions with faculty, the interviews offered a new opportunity for acquisitions and technical 
services librarians from the Denver campus and Health Sciences Library to liaise directly with patrons. 
For instance, during interviews at the Anschutz Medical Campus faculty members expressed an 
unremitting need to be knowledgeable about the most recent developments in their field, focusing 
predominantly on research published in journals during the last year. The critical requirement for 
immediate notification of the latest publications was revealed when one faculty member shared that he 
subscribes to table of contents alerts for more than 25 journals. Study participants consistently 
articulated their appreciation for the focus on current and online journal content available through the 
Health Sciences Library. They valued anytime and anyplace access to the most essential research titles, 
such as the Journal of Orthopedic & Sports Physical Therapy, for their research and teaching. 

While expressing their preference for immediate access to the most current journal holdings, the 
Physical Therapy faculty members conveyed their understanding of budget limitations. They did not 
expect the Health Sciences Library's collection to include backfile access or subject areas outside of 
medical and allied health. Study participants said that they were satisfied with relying on interlibrary 
loan (ILL) for older articles and periphery topics, despite the Health Sciences Library being the only 
library within the CU System that operates ILL as a cost-recovery unit by charging faculty for services. 

Similarly, the Denver Chemistry faculty expressed high praise for Auraria Library's collections and 
services. When asked for suggestions to improve the collection, they consistently commented on their 
satisfaction with the library's subscriptions to core titles in their field. However, a few months after the 
ROI interview, one study participant emailed Pan with a request for greater access to American 
Geophysical Union (AGU) journals. Coincidentally, the library had just received the renewal and an offer 
from the publisher to upgrade from a few titles to the complete complement of AGU titles. Knowing the 
faculty member's need for these journals was a contributing factor for the Denver campus to make a 
decision to contribute to a CU system consortia purchase of the AGU full electronic journal package. 

At the Health Sciences Library, Williams received requests for consultation services on copyright when 
study participants asked questions such as "can I distribute a copy of an article from my personal journal 
subscription to students?" and "if I give the library a reading list, can I get those texts put on reserve?" 
Both Williams and Pan had a chance to deliver one-on-one instruction on how to use the library website 
and clarify the library's buyer role in providing access to propriety publisher content. On the Boulder 
campus, Fong and Wiersma answered queries about access to specific resources and the liaisons, who 
attended interviews, were able to learn more about the research projects of the faculty participants. 
Several faculty members conveyed a vague notion that the library provided access to journals on- 
campus. They welcomed further explanation on how to achieve off-campus access to library databases 
and journals via Google Scholar. In addition, the interview process also inspired new research 
collaboration at the Denver campus. Although a Chemistry teaching instructor did not publish and 
therefore could not participate in the CU ROI study on faculty outcomes, she was very interested in the 
project. Eventually she and an Instruction Librarian collaborated with Pan to develop a separate case 
study to quantify the benefits Chemistry students receive in their education by using library collections. 

In hindsight, the initial phase of the CU ROI model was very challenging. Identifying, recruiting, 
scheduling, and interviewing study participants created logistical obstacles and was more time 
consuming than anticipated. Despite these efforts, the qualitative method provided valuable insights 
and richer context for understanding faculty needs. The researchers successfully gathered details on 
study participants' publications and use of library materials. By re-conceptualizing the interviews as an 
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outreach opportunity, however, a ROI study can generate unexpected learning and teaching moments 
that establish or improve communication and relationships with a department and individual faculty 
members. 

Phase 3: Calculating ROI 

Using data gathered from faculty interviews and curriculum vitae, the authors derived the ROI value of 
library collections used to support faculty research, i.e. scholarly articles written by pilot participants. As 
a result, the CU ROI model shows the percentage increase in value on dollars spent by the library (in 
terms of journal subscriptions) to achieve a benefit for faculty (derived from potential savings from the 
average cost they would have had to pay to buy articles directly from publishers). Furthermore, the CU 
ROI model, using cost benefit analysis, identifies the return benefit academic faculty received for every 
dollar the library spent on journal subscriptions. 

The CU ROI model is based on cost benefit analysis (CBA) and return on investment (ROI). CBA is the 
ratio showing dollar value of benefits gained for dollar value of costs as represented by the following 
formula: 

Benefits -r Costs = CBA 

ROI uses the same values as CBA, however, ROI is shown as a percentage. It conveys the return or 
increase in value on dollars spent to achieve a benefit. The formula is: 

{{Benefits - Costs) -r Costs) x 100 = ROI 

To calculate the benefits and costs, the CU ROI model (see Table 1) applies a combination of data 
obtained from faculty interviews, citation analysis, and costs. Library costs are the library's expenditures 
to purchase online journal subscriptions for the references cited in faculty publications. Faculty benefits 
are defined as the average price that faculty would have had to pay in order to buy their cited articles 
directly from publishers (Pan, Williams, Wiersma, & Fong, 2012). 



Table 1: The CU ROI model 





Variable 


Data Source 


A 


# of published articles in 1 year 


CV 


B 


# of all cited references 


Citation 
Analysis 


C 


# (%) of cited references from journals 


Citation 
Analysis 


D 


# (%) of cited references from online subscriptions 


Citation 
Analysis 


E 


# of additional articles read (but not cited) 


Interview Ql 


F 


# of additional articles read from journals (but not 
cited) 


E x C% 
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G 


# of additional articles read from online journals _ 

. ,, F x D% 


H 


# of citations and articles read (but not cited) from D + G 
online resources 


Average cost per article (random sampling of ... 



J 


Faculty benefit - average cost to buy articles 
directly from publisher with pay-per-view 


HXI 


K 


Library costs - online journal subscriptions for 
cited references 


$$$ 


ROI 


Percentage showing return or increase in 
value on dollars spent (by library) to achieve a 
return (for faculty) 


J-K/KxlOO 




Ratio showing dollar value of benefit gained 







Faculty responses to interview questions were collected and applied to the appropriate variables. For 
example, from the first interview question ("In your most recently published article, for every article you 
cited how many additional articles did you read but did not cite ?") the number of additional articles 
read but not cited were determined (variable E). 

Study participants' most recent research publications were identified from the curriculum vitae. The 
authors limited the scope to articles published within two years of the study which made the citation 
analysis more manageable for the pilot study. After obtaining copies of the faculty's articles, the source 
of full-text access for each journal article cited was identified. Based on the cited references in each 
article, three variables were calculated: the number of all cited references (variable B); the percentage 
of journal citations (variable C); and the percentage of journal citations from online library resources 
(variable D). 

After the interviews and citation analysis were finished, the authors calculated library costs. The library 
costs were based on the prices of subscriptions paid by each library for journals cited by the study 
participants (variable K). 

The final variable needed to measure ROI/CBA is the faculty benefit or a monetary value for the cost of 
having access to library collections (variable J). Poll and Payne (2006) explain "as most library services 
have no equivalent on the common market and therefore no 'market prices' can be determined, two 
other ways have been tried for assessing an economic value: assessing time costs (replacement value of 
a client's time) and using the contingent valuation method" (p. 554). 

In most cases, contingent valuation constructs a market by surveying current and potential library users 
to determine their willingness-to-pay (WTP) and willingness-to-accept (WTA). Poll and Payne (2006) 
describe WTP as "what would you pay for maintaining this library/this special library service" and WTA 
as "which sum would you accept as an equivalent if this library/this special service were given up" (p. 
554). For example, Tenopir and King (2007) used contingent valuation to estimate the implied value of 
journal collections. Faculty survey participants were asked "to indicate what they would do if the source 
they used for their last reading (e.g. library collection) was not available to them. Then [they asked] 
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them to speculate what it would cost in terms of money or time to get the same information if the 
source they used was not available to them" (Tenopir and King, 2007, p. 204). 

Faculty members have many options for obtaining materials they need for their research. These 
alternatives could be free or have a cost. For example, they could be borrowing materials from a 
colleague (free), obtaining a copy from the library or Interlibrary loan services (free at the Boulder and 
Denver campuses but a fee-based service at the Anschutz Medical Campus), or paying for an article 
directly from a publisher. Instead of relying on a subjective value for faculty WTP, the CU ROI model 
adapted the contingent valuation method to simulate a market value for library collections. 

The CU pilot study used a random sample of journal titles cited in faculty publications to arrive at an 
average price per article that a faculty member would have to pay in order to obtain an article directly 
from the publisher. For example, at the Denver campus, study participants cited 158 journals. Pan 
randomly selected 15 journals (10% of 158) that were available from the publishers' pay-per-view 
services. Using the pay-per-view article price for each of these 15 journals, Pan calculated the average 
cost per article for her random sample to be $34.22 (variable I). This amount was multiplied by the 
number of citations and articles read, but not cited, from online resources (variable H) to calculate the 
faculty benefit (variable J). 

Phase 4: Analyzing Results 

After gathering and applying data to the CU ROI model, the authors were able to calculate ROI results 
for each campus. For every dollar invested in journal subscriptions, using the average pay-per-view 
article price, the CU ROI model produced a positive return of 66% and 144% for Denver and Boulder 
respectively, and a negative return of -19% at the Anschutz Medical Campus (see Table 2).The 
researchers calculated low and high pay-per-view article prices based on standard deviation from the 
average pay-per-view article price. With standard deviation, the authors are 95% confident the average 
price falls somewhere within this range. For example, for CU Denver, a low pay-per-view price of $31.94 
results in a 55% ROI while a high pay-per-view price of $36.51 results in a 77% ROI. In other words, CU 
Denver still had a positive ROI regardless of the low and high article prices. The outcomes from CU 
Boulder, Denver, and Anschutz Medical campuses are summarized in Table 2 (Pan, Williams, Wiersma, & 
Fong, 2012, p. 15). 

While these ROI figures were informative, they were not generalizable for each institution given the 
small number of faculty who participated in the study. These inconclusive findings could have caused 
the researchers to feel unsatisfied. By conducting interviews with, examining references cited by faculty, 
and researching costs for journal subscriptions and articles, however, they were able to discover new 
and surprising perspectives on faculty needs. 
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Table 2: Pilot Results for the University of Colorado 





Average 


Low Standard 


High Standard 




Article Cost 


Deviation 


Deviation 


CU Boulder 








cost per article 


$32.53 


$29.66 


$35.41 


CBA 


$2.44 


$2.05 


$2.44 


ROI 


144% 


105% 


144% 


CU Denver 








cost per article 


$34.22 


$31.94 


$36.51 


CBA 


$1.66 


$1.55 


$1.77 


ROI 


66% 


55% 


77% 










cost per article 


$36.36 


$30.44 


$42.27 


CBA 


$0.81 


$0.68 


$0.95 


ROI 


-19% 


-32% 


-5% 



There are many factors that could shift ROI from positive to negative including library journal 
subscription costs, faculty productivity, and the cost per article from publishers' websites. Initially, one 
may conclude that the resulting ROI value could be used to signify whether or not the library's collection 
development strategy is supporting a department. For example, the Denver campus had a positive ROI 
which could indicate that the library is providing access to the majority of the resources needed to 
support faculty research for the Chemistry department. In contrast, the Anschutz Medical Campus had a 
negative ROI for the Physical Therapy department, which could suggest that the library has gaps in the 
collection or is subscribing to resources that are not getting optimal use by faculty. However, because 
results varied significantly from between campuses and year to year, the authors realized that ROI is still 
just one metric that needs to be placed within the overall context of the data collected and used in 
conjunction with other collection assessment metrics. Pan, Wiersma and Fong (2011) recognize that 
"like any statistical measure, ROI and CBA calculations are only meaningful when put into context" (p. 
460). 

The results of the CU ROI pilot study provide further evidence that multiple measures are needed to 
demonstrate library value. "The benefit of multiple methods is that numbers in and of themselves 
rarely tell the full story. Interviews and surveys allow the faculty to tell their story of how they use the 
library in their grant proposal process, in teaching, and in their research and work life" (Tenopir, 2010a, 
p. 46). In addition to the ROI calculation, the mixed methodology approach of the CU ROI model 
provides data that could be used to inform collection development decisions and strategies. 

For example, pairing citation analysis data with feedback from faculty interviews could be used to 
inform decisions regarding renewals and cancellations. Libraries often seek input from campus 
colleagues before cancelling journal subscriptions or moving materials offsite. Faculty members 
typically provide input about core journals that represent the research interests of the entire 
department. However, it can be difficult to identify niche publications that are important to specialized 
research within the discipline. During the interviews at the Boulder campus, study participants often 
mentioned relevant journals in their field as well as resources that were important to their particular 
research. Citation analysis confirmed the importance of many core journals within each field and also 
identified journals that were necessary for individual faculty research. Both qualitative and quantitative 
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methods provided valuable feedback about journals that would likely be candidates for cancellation 
based solely on cost, usage, or subject coverage. 

By the same token, the model could provide an alternative metric for cost per use calculations and 
usage statistics. Cost per use calculations and usage statistics indicate that the library's existing 
collection is being accessed. They cannot identify, however, the significance of that use. By combining 
citation analysis and usage data, detailed analysis at the Denver campus revealed that some low use 
journals actually resulted in a high number of faculty citations. One of the low usage journals (18 full- 
text articles accessed) was cited at a greater rate than the highest ranked journal and most accessed 
journal within the field. This title might have been a candidate for cancellation based on usage alone, 
but the added citation rate information provided additional context for measuring the value of this 
journal (Pan, Wiersma, & Fong, 2011). 

The CU ROI model could provide a framework for evaluating overall collection development strategies 
and provide evidence to support the investigation of alternative acquisition models. As a result of the 
negative findings, the Health Sciences Library began to analyze their collection development policy and 
explore options to improve the collection and services. Since 2005, the Health Sciences Library has been 
increasingly transitioning from print to electronic resources in response to a campus LibQual survey. 
Today 98.5% of the materials budget is dedicated to continuing online materials with annual 
subscription fees. After several years with no or minimal increases, the materials budget is failing to 
keep pace with inflation despite the use of common strategies for controlling costs such as negotiating 
multi-year deals with annual price caps and participating in consortial purchases (Williams, Pan, 
Wiersma, & Fong, 2012). 

From a detailed analysis of the journals that were cited by faculty, Health Sciences librarians discovered 
that the library was providing access to the core journals for the discipline but the cost of those journals 
was relatively high compared to resources in other disciplines. To make matters worse, in 2012 serials 
prices increased six percent versus five percent in 2011 while "the Consumer Price Index advanced 2.9 
percent for 2011, which means serials inflation continues to far exceed general inflationary pressures... 
Prices for science, technology, and medical (STM) serials remain the highest, compared with prices for 
serials in other subject areas..." (Bosch & Henderson, 2012, On the upswing: serials prices para. 4). 

In the "new normal" economic realities, the Health Sciences Library's negative CU ROI results 
foreshadow the future for many academic libraries - flat or decreased budget in stark conflict with high 
renewals - and reinforce concerns regarding the sustainability of the existing models for purchasing 
electronic content. At the Health Sciences Library, the collection development conversation changed 
from undertaking a typical journal cancellation process to meet budgetary goals to considering 
alternative acquisitions and access models. Should a library forgo the greater access provided by big 
deals in favor of smaller deals that provide subject specific collections? Should the library negotiate 
lower prices in exchange for current access and for relinquishing perpetual and archival rights? Should 
the library explore pay-per-view models and patron-driven acquisitions? At the present moment, the 
ROI results inspire more questions and definitive answers remain to be determined. 

Conclusion 

The CU ROI pilot study developed and tested a theoretical model for measuring the impact or value of 
library resources in faculty research. While each campus applied the same protocols, the findings of the 
study provided institutional insights that were meaningful to each institution's mission. With this in 
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mind, the researchers discourage definitive assertions or interpretations of their pilot study results. 
Although the final ROI figures were informative, they are not generalizable for each institution given the 
small number of faculty who participated in the study. However, if applied appropriately, the CU ROI 
model has the potential for demonstrating how library collections support research outcomes or, 
alternatively, results can support a transition to new acquisition and access strategies. 

The process of collecting and examining data to calculate a ROI value provided unexpected benefits 
including relationship building with academic departments, increased awareness of resource needs, 
promotion of library collections and services, and evidence to support the investigation of alternative 
acquisition models. In the final analysis, the study was valuable for more than the number that was 
derived as ROI. Although the study required a great investment of time and careful deliberation and 
analysis, the process of collaborating with colleagues at other campuses and engaging faculty in the 
research was rewarding. The interviews were an excellent outreach opportunity. These one-on-one 
conversations provided an opportunity for in-depth discussions about collections relevant to faculty 
research and elicited information about faculty member's preferences for formats, interlibrary loan, and 
access to resources via the libraries' catalogs. Moreover, the added benefits of outreach and learning 
about faculty research are incentives that the authors feel outweigh the challenges of implementing a 
ROI study. 
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